Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks a lot for prioritizing it.
docs/source/en/optimization/fp16.md
Outdated
|
|
||
| [Kernels](https://huggingface.co/docs/kernels/index) is a library for building, distributing, and loading optimized compute kernels on the [Hub](https://huggingface.co/kernels-community). It supports [attention](./attention_backends#set_attention_backend) kernels and custom CUDA kernels for operations like RMSNorm. | ||
|
|
||
| The [Diffusers Pipeline Integration](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/references/diffusers-integration.md) guide shows how to integrate a kernel. Create a custom optimized attention processor, patch all modules in the model, and inject the kernel into the pipeline. |
There was a problem hiding this comment.
The kernel skill basically lets users get an agent to write custom kernels for a model and hardware. It's not specific to the attention processor but also other modules as well such RMSNorm. Should we make it clearer?
There was a problem hiding this comment.
lmk if this is clearer!
docs/source/en/optimization/fp16.md
Outdated
| > [!TIP] | ||
| > Install the [add cuda-kernels](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/SKILL.md) skill to teach Claude or Codex how to write a kernel. The [Custom kernels for all from Codex and Claude](https://huggingface.co/blog/custom-cuda-kernels-agent-skills) blog post covers this in more detail. | ||
|
|
||
| For example, a custom RMSNorm kernel with [torch.compile](#torchcompile) speeds up LTX-Video generation 1.43x on an H100. |
There was a problem hiding this comment.
It wasn't just RMSNorm but also other modules implemented with custom kernels.
There was a problem hiding this comment.
i mention RMSNorm as an example only for the benchmark results below
There was a problem hiding this comment.
Thanks! I would like to also see what @burtenshaw thinks about this.
3 participants
adds a kernels section in the
Accelerate inferencedocs with the results:Attention backendsdocs which demonstrates support for loading attention kernels withset_attention_backend